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Abstract 

We give a formalization of tlie notion of test purpose based on (suitably re- 
stricted) Message Sequence Charts. We define the validity of test cases with re- 
spect to such a formal test purpose and provide a simple decision procedure for 
validity. 



1 Introduction 

The quality of a test system directly influences the quality of the tested implementation: 
high quality test systems are essential to obtain high quality implementations. Hence, a 
common problem in the testing area is the so-called "test the tester" problem 1 12 1: how 
can the validity of a test system with respect to a given specification, and therefore the 
quality of the test system, be assured? To put it in conformance testing terminology: 
how can it be assured that a test case achieves its test purpose? 

One approach used to obtain valid test systems is the derivation of test cases from 
formal specifications or test purpose definitions. Other approaches focus on the man- 
ual or automated simulation against a formal specification (see 1 14 1 for a description 
of tools that employ these two approaches). While many modern telecommunication 
protocols come with (semi-) formal specifications of test purposes, a formal protocol 
description is provided only in very few cases (see |5| for a notable exception). For 
example, Internet Protocols defined in RFCs use natural language to define the seman- 
tics of the specification. Due to this, a formal description of the specification would 
have to be elaborated to allow for an automatic generation of valid test cases. Addi- 
tionally, even if formal descriptions are available, automated generation methods only 

*This paper appeared in the proceedings of the 22nd IFIP WG 6.1 International Conference on Formal 
Techniques for Networked and Distributed Systems (FORTE 2002), number 2529 Lecture Notes in Computer 
Science. 
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generate test skeletons that need to be manually refined to obtain executable test cases 
for the execution against a concrete system implementation. For all these reasons, the 
implementation of test cases is still performed mainly in a manual manner 

In this paper, we give a new answer to the "test the tester problem", namely, to 
check the validity of a (possibly hand-written) test case against a formal test purpose 
definition. It does not rely on the existence of a formal description of the system under 
test (SUT) or the test system, but requires a formally defined test purpose. From this test 
purpose, the allowed and required behavior of the test case is derived. This information 
is then used in a guided simulation of the executable test system to determine whether 
the test system is valid with respect to this test purpose. Since our approach is solely 
based on test purposes, it is not necessary to develop a complete formal specification of 
the system as test purposes are only a partial description of the system. We use Message 
Sequence Charts (MSC) as the formal test purpose description language |6|, which is 
widely used in the system development process in the telecommunication area. This 
allows for an easy re-use of the uses-cases developed during system design as a solid 
basis for the test purpose definition. This further reduces the work necessary for the 
test purpose specification. 

Despite the fact that MSCs are widely used to capture test purpose, theoretical 
studies of MSCs so far seem to have failed to address the following issues: 

• What does it mean for a test case to implement a test purpose, i.e., when is a test 
case valid w. r. t. a test purpose? 

• When is an MSC a well-formed test purpose, i. e., when does an MSC character- 
ize behavior that is indeed (black-box) testable? 

We address these issues using a semantics for MSC based on pomsets fl 1'^ in the 
spirit of 1 8 1. We then describe a simple decision procedure for the validity of test cases 
w. 1. 1. a test purpose and prove its correctness. 

The paper is structured as follows: Section|2lof this paper introduces the partial or- 
der semantics of MSCs and their usage as formal test purposes. In Section|3j we define 
test case validity, describe the decision procedure and prove its correctness. Section|4] 
presents one possible implementation design for an MSC based test validator Section 
Isjconcludes. Proofs of key lemmata and theorems can be found in the appendix. 

2 Formal Test Purposes 

To check (or even define) validity of a test case wrt. a test purposes, we need a formal 
definition of a test purpose together with suitable semantics. In this section, we sug- 
gest a formalism to formally express test purposes and establish a set of criteria that 
guarantee that a test purposes indeed describes (black-box) testable behavior 

We use Message Sequence Charts (MSCs) to express formal test purposes because 
they are widely used to capture test purposes and semantics based on different ap- 
proaches are available. We have chosen semantics based on pomsets [4. .11 J in an 
adaption of the definition of JS] to better suit our purposes. The particular choice of 
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Figure 1: Example MSCs. 



semantics of MSCs in not central to our approach, but obviously some choice has to be 
made. Using the more operational semantics from ||9]0 would lead to similar results. 

After a short overview on the employed MSC syntax for test purposes, we reca- 
pitulate the pomset-based semantics of MSC and define when an MSC constitutes as 
well-formed test purpose. 



Message Sequence Charts. The MSCs in Fig. serves as an explanatory exam- 
ple for the basic MSC language as used throughout this paper The most fundamental 
constructs of MSCs are instances and messages. Instances represent components or 
communication interfaces that exhibit a sequential behaviour. Our example MSC mi 
consists of three instances p, q, and r. A message exchange between a sending instance 
p and a receiving instance q comprises two events lp,qa and Ip.qU for sending the mes- 
sage a at p and for receiving a at q, respectively. Graphically, messages are depicted 
by arrows between instances labeled with messages. 

Events are considered to be causally or temporally ordered only if they are located 
at the same instance (in this case the ordering is top-to-bottom), or if they are part of the 
same message exchange. In our example mi, the event lp,qa precedes the events ?j,,ga 
and lp,rb, but no assumption on an ordering of the events Ip ^b and Iq ^c is expressed, 
even if !p is drawn above Iq^rC- 

There is a way to express the concurrency of events of the same instance: the 
concurrent region (coregion, for short). Coregions are depicted by dashed sections 
on the corresponding instance line bordered by small horizontal bars: the events that 
occur on this dashed section are supposed to happen in parallel. In our example, the 
events ?p,r& and Iq.rC are temporally unrelated. On the other hand, it is possible to use 
general order arrows (dotted lines between events with an arrow head in their middle 
section) to express causal orderings of events on different instances. In mi, the event 
!ij,rC precedes Ip^gd. Finally, the MSC language allows to express message exchange 
with the environment of a MSC; e. g. in mi the message e is send to the environment 
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of this MSC. 

The MSC formahsm provides not only communication primitives but also control 
structures. For our purposes, only the alt operator, modeling nondeterministic choice, 
is of importance. m2 in Fig.^shows an example: A choice between sending a fromp 
to q and sending b from g to p is expressed. A final construct considered in this paper 
is that of conditions. Conditions model global states or predicates related to more than 
on instance; TO2 contains two conditions Ci and C2. It is not an easy task to assign a 
formal meaning to conditions. However, we use conditions only to express test verdicts 
and handle them formally in a special way. We will discuss this topic in detail in a later 
section. 

Other important concepts of the basic MSC language not covered in this paper 
are: loop inline expressions (since tests are finite, loops occurring in test purposes 
comprises alway finite, fixed boundaries and therefore can be unfolded), and especially 
timers, which require extra considerations and will be dealt with in forthcoming work. 

Expressing test purposes. We will use the MSC formalism to capture test purposes 
in the following way: the set of instances is partitioned into a non-empty set of port 
instances and a non-empty set of SUT instance. Intuitively, the port instances represent 
the different ports (PCOs, interfaces) at which the SUT interacts with its environment. 
Conditions that span the port instances are used to assign the test verdicts. 

The SUT instances are used as "syntactic sugar" and serve two purposes: (1) as 
communication partners for the port instances, and (2) to impose an ordering of the 
sequence of messages. The same could be achieved by using communication with 
the environment and generalized orderings, but our approach leads to a more concise 
and intuitive representation of the test purpose and matches the common usage. Fig. |2] 
shows the two alternative ways of depicting a simple test purpose: after having received 
the message a on both its ports p and q (in arbitrary order), the SUT answers by sending 
the message h, again both on port p and q. If the message is sent on port p before it is 
sent on port q then the SUT shall pass the test, otherwise it shall fail. We will come 
back to this example later in this paper. 

2.1 Partial Orders 

We quickly recapitulate how pomsets can be used to assign a semantics to MSCs. We 
start by introducing the basic notations used throughout this paper. 

To avoid tedious notation, we fix the following convention: if a structure S = 
{A, B, . . .) is introduced, the components of S will be denoted by As, Bs, ■ ■ ■ 

For some set A, 'J'{A) is the set of all subsets of A. For R C Ax B and a G A, we 
denote the image of a under R by R{a) =df {b ^ B : a R b}. For C C ^ we define 

RiC) =df Ua6c5«)- 

The inverse R ^ of a relation R, the identity relation id^ on A, the relational com- 
position R ■ S OT two relations R, S, the transitive closure i?+ of R, and the reflexive- 
transitive closure R* of R are defined in the usual manner. 
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Figure 2: Expressing the same test purposes with and without SUT instances 



Lposets. For the rest of this paper let us fix a finite alphabet S. A labeled partial 
order (Iposet, for short) over S is a structure x = {E, <, A) where _E is a finite set 
of events, < C iJ x £' is an (irreflexive) partial order, and A : _E ^ S is a labeling 
function. 

Let x be a Iposet and let ei, 62 e E^. We use the following notions: The reflexive 
closure of <x is =df <a;U idfi^. Unrelated events are called concurrent, i.e., 
ei cOa; 62 Odf ei 62 & 62 61, whilc related events are in line: ei li^; 62 <^df 
ei <a; e2 V 62 <x ei. 

The downward closure of a set Z? C iJ^, is 4a;_D =df ^x^{D). If D = l^D holds, 
then D is called downward closed in x. By C(a;) we denote the set of downward 
closed sets in x. If D <Z E^, then x[D] =df {D, <x H (D x D), A f is the Iposet 
generated by D in x (X \ D denotes the restriction of A to D). 



Pomsets. Lposets x and y over S are called isomorphic, written x = y,if there is a 
bijection f : E^ ^ Ey such that (ei <x 62 <^ /(ei) <y /(e2)) & Aj; — XyO f holds. 
A partially ordered multiset (a pomset for short) over E is an isomorphism class of 
lposets, i. e., a set [x] =df {y : x = y}. We fix the convention, that pomsets are denoted 
by boldfaced small letters x, y, z. Moreover x is assumed to be the equivalence class 
[x] of x. By this convention, E^. always denotes the set of events of a representative x 
of X. The class of pomsets over S is denoted by P(S). 

Fig. |3lshows examples of pomsets. Graphically, we represent pomsets as directed 
acyclic (not necessarily connected) graphs. Nodes are labeled with elements from the 
underlying alphabet E. Transitive arcs are sometimes omitted. 
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Let x,y £ P(S) be pomsets. Then x is called a prefix of y — denoted x ^ y — iff 
there are representatives x ^ x and y £ y such that Ey h C(x) C C(y) holds. 

If there are representatives x ^ x and y E y such that E^ — Ey k, <x <y holds, 
then X is called less sequential than y. This is denoted hy x ^ y. It is easy to see that 
both ^ and =^ partially order P(S). In Fig.|3] x ^ y, x ^ z, and y ^ z, holds. 

An alternative definition of the prefix relation ^ and the ordering by the degree 
of sequentiality =:<; can be obtained by introducing the notion of weak homomorphisms 
between representatives of pomsets (2|. 

Special pomsets that will be encountered in this paper are: 

1. Letters a = [{a}, 0, a a] for a G E (we abuse a,b,c, . . . to denote both letters 
fromP(S) and from E). 

2. Strings [{0, . . . , n — l},<,i i-^ Oj] for aoai . . . a„_i G E*, where < denotes 
the standard order relation on integers. 

3. The empty word e = [0, 0, 0]. 

In this paper we do not distinguish between strings and pomset strings, i. e, if E is an 
alphabet then E* is considered to be the set of pomsets a over E such that is a total 
ordering. 

If a; e P(E), then by lin(a;) =df {tr € E* : a; ^ cr} we denote the set of 
linearizations of x. 

Dependencies and Weak Sequential Composition. A reflexive and symmetric re- 
lation D C E X E is called a dependence on E; for the rest of this paper let Z? be a 
dependence on E. If x and y are Iposets over E, such that E^ r\ Ey ~ ^ holds, then 
the weak sequential composition x o^j y is defined by 

xooy =df [E^UEy,{<^U<yUR)+,\^U\y], 

where R C E^ x Ey is given by ei R 62 -^df ^x{ei) D Ay (62). 

A pomset x is called D-consistent if we have, for all ei, 62 G E^, ei cOa; 62 ^ 
^Xx{ei) D Xx{e2)- Let P(E, D) denote the class of /^-consistent pomsets. Clearly, if 
X and y are D-consistent, then also a; o^, y is. 
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Figure 4: An annotated MSC. 



Another operation on pomsets which is closely related to oj^ is the unsequential- 
ization via 13: If a; e P(S), then by {x) jj we denote the pomset [E^, R^, Xx], where 
R C Ex X Ex is defined by ei R 62 ^df ei < 62 & A2;(ei) D Xxie2)- ' 

The pomsets in Fig. |3lare all /^-consistent for the dependence D shown in that 
figure. We have {z)jj = y. 

The following lemma justifies the relation between the operations o^, and 

Lemma 1 Letx,y in P(S). Then {xo£iy)jj = {x)jj oq {y) jj. 

Some more definitions: if ^ C S is a set of symbols and a; is a pomset, then 
X \ A =df [3^[Ak ^ (^)]]- X \ A\s called the restriction of x to A, i.e., x restricted 
to those events labeled with elements from A. Finally, a set of pomsets X C P(E) is 
called pre-closed ifx^XSzy^x^yeX holds. 

2.2 Partial Order Semantics for MSCs 

To define the semantics of MSCs based on pomsets, we first need to fix an alphabet Sc 
and a dependence on Sc- 

Communication Alphabet and Dependence. Let M be a set of messages and P a 
set of instances fixed throughout this paper We assume that there is a non-empty set 
r C P of pori instances; the instances in P — T will be called SUT instances. Usually 

'The operations o^i and (■) impose an interesting and fruitful connection to tlie theory of Mazurkiewicz 
traces 1101 . Although it is far beyond the scope of this paper it should be noted that pomsets of the 
form (x) j~, S P(S, D) are just alternative representations of Mazurkiewicz traces: in fact we have that 
lin((a;)^) is a Mazurkiewicz trace over S and D; moreover, the operation ojj coincides with trace concate- 
nation. 
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we will have \P — T\ = 1, but the theory presented in the following does not rely on 
this. 

Let El , E? be the two alphabets: 

1. El — df {!} X P X M X P is the set of send actions. Its elements {\,p, m, q) will 
be denoted by Ip^qin, 

2. E? =df {?}xPxMxP is the set of receive actions. Its elements (?, p, m, q) S 
E? will be denoted by 7p_qm. 

We put Ec =df El U E? to be the set of communications. The mapping tc(a) 
identifies the instance of an action a € Ec, i. e., Lci^.p.qin) —di P and Lc{lp.qm) —di q. 
We put Eo =df {a £ Ec : tc(a) £ T} to be the set of tester observable actions. For 
convenience, we furthermore define E^ =df So H Ei and E^ =df Eq H E?. 

Fig. |4]gives a few examples of this syntax of actions. It shows the expansion of the 
first alternative of TO4 where the generalized ordering has been replaced by sending the 
void message 0. The messages have been annotated with the corresponding symbols 
from Ec. 

To build pomsets from actions, we define the dependence on Ec: let Q 
Ec X Ec be the smallest reflexive, symmetric relation containing: 

• (a, b) with ic(a) = Lc{b) and a and b are not placed on the same co-region, 

• {Ip^qin, Ip^q-m) for instances p,q E P and messages m G AI. 

To keep things simple, we restrict ourself to the following MSC operators: message 
sending and receiving, co-regions, and the alternative inline expression, which allows 
the expression of optional behavior and finite iterations. We simulate general ordering 
by sending a void message 0, which might also be sent between two port or SUT 
instances. Conditions are only allowed to assign verdicts and are not dealt with by the 
semantics. In order to obtain a set of Z3c -consistent pomsets, we have to impose the 
restriction that identical actions (e. g. sending of a message twice from an instance p to 
an instance q) are not placed on the same co-region. 

The semantics of an MSC M is given by a pre-closed set of pomsets Xm C 
P(Ec, Dc). We illustrate the construction of Xm only by informal means of an ex- 
ample (ma from Fig. |3; the translation is done similar to L8J with slightly different 
syntax for events of pomsets. 

In the following, Oc abbreviates o . 

The semantics of our example 771,3 is given by the set X„i3 : 

Xrns = e P(Sc, -De) : 2; < a; Oc V 2; < a; Oc y^}. 
where x, y^, and j/j are defined by: 

X df -pjT^ ^-q^r^ ^p,r^ ^q,r^ (1) 
Vl =dl K.pbOclr^pbOclp^qOo^lp^qOo^l^gbo^l^gb (2) 
2/2 =df !r,<;& Oc ?r,g& Oc !<;,pO Oc ?g,pO Oc !r,p& Oc ?r,p& (3) 

Without a proof (which would require a more formal treatment of the definition of 
Xm we state: 
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Lemma 2 IfM. is a MSG, then (x) = xfor all x G X^. 

2.3 Message Sequence Charts as Test Purposes 

Now that we have explained how to assign semantics to an MSC, we show how MSCs 
can be utilized as a formal language to express test purposes. We discuss how the 
notion of a test verdict can be integrated into an MSC and how it can be guaranteed 
that an MSC specifies behavior that is amenable to black-box testing. 

Verdict assignments. Syntactically, a verdict assignment is expressed by a condition 
on the port instances on the very end of each terminal alternative of the MSC. Semanti- 
cally, the condition-like constructs pass, fail, and inconc are not treated as an ordinary 
condition but as a convenient way to define a verdict assignment: ^ 

Let V =dt {pass, fail, inconc, none} be a set of verdicts and let V[ ~df V^— {none} 
be the set of final verdicts. A mapping v : X ^ V for some finite, pre-closed set of 
pomsets X is called a verdict assignment to X if, for all a; G X, we have: 

1. 3y G X.x ^ y & v{y) =|= none, i. e., every pomset can be extended to a pomset 
that is assigned a final verdict, and 

2. v{x) =1= none =^ Vy G X.x <j: y, i. e., pomsets that are assigned a final verdict 
are maximal in X. 

The verdict conditions drawn in an MSC M are used to define a verdict mapping 
vm- Again, we introduce this informally by the example of ma from Fig. |2j where Vmi 
is defined by: 



It is obvious that not every MSC that satisfies the syntactic restrictions that have 
been introduced above constitutes a test purpose, i. e., describes behavior of the SUT 
that can be tested in a black box testing approach. For example consider a modification 
of TO3 from Fig. 12] where the generaUzed ordering constraints have been eliminated. 
There the verdict does not depend on the order in which the messages b can be observed 
at the ports of the SUT but rather on the (SUT-internal) events that cause these messages 
to be sent. Clearly, such an event is not visible to a black-box test system and hence no 
test case can distinguish between the behavior of the first and second alternative. In the 
following we present a number of criteria that an MSC must satisfy to be considered 
a well-formed test purpose. Later we will see that these criteria indeed guarantee the 
existence of a valid test case for a test purpose. 

^Alternatively, one could allow verdict conditions to appear also at other places within the MSC and, e.g., 
use the verdict assignment rules of TTCN-3 [3J to resolve the case where different verdicts are encountered 
during a single run through the MSC. 




pass, \i Z = X o^ y^] 
fail, \f z = X o^y^\ 
none, otherwise. 
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Well-Formed Test Purposes. First, we define a function that reduces the semantics 
of an MSC to the information that is available to the test case, i. e., the sequences of 
events that occur on port instances: 

Given test purpose MSC M with semantics Xjy[. For x G Xjy[, we define the tester 
observable traces of x by obs(a;) =df lin(a; \ So). 

A MSC M is called a well-formed test purpose if it is possible to determine its state 
(and hence assigned verdict) based on this information in its tester observable traces, 
i. e., if 

WFi. for every x,y ^ \ Eq, lin(a;) n lin(y) ^ implies x = y. 

Unfortunately, this restriction does not yet suffice to guarantee that an MSC de- 
scribes testable behavior. Another aspect that needs considerations is which party re- 
solves essential choice in the sense of the following definition: 

Let X C P(Ec, -De) be a pre-closed set of pomsets. A pomset x ^ X is called a 
choice point for two actions a, 5 £ T,c in X if x a ^ X , x b e X , and 

{y e max ^{X) : X a y} =j= {y e ma.x ^{X) : x b y} , 

where max^ {X) denotes the ^-maximal pomsets in X. 

Coming back to example from Fig. |2| with semantics X^^ as defined in Q- (|3}, 
X I" Eo is a choice point for Ir.pb and Ir.qb- On the other hand, e is not a choice point 
even though there are two "available" communications, namely Ip^rO, and Iq^rd, since 
this choice does not alter the reachable maximal configurations. 

We require, for a well-formed test purpose, that each choice point is resolved by a 
message from the SUT: 

WF2. If a; is a choice point of Xj^i \ Eq for actions a, 6 G Eq, then both a,b £ E^. 

This restriction is necessary because both other possibilities for a choice point 
(a, 6 G Eq or a G Eq and 6 G E^) are undesirable in a test purpose: a choice that 
has to be resolved by the test case indicates that the test purpose should indeed be (at 
least) two test purposes, one for each choice of the test case. Otherwise, a determinis- 
tic test case will only be able to test the part of the test purpose that corresponds to the 
(necessarily fixed) way the test case resolves the choice. On the other hand, a choice 
that can be resolved simultaneously by the test case and SUT leads to problems be- 
cause it might lead to a race condition where both test case and SUT resolve the choice 
in an inconsistent manner. This situation bears strong resemblance to the presence of 
non-local choice in the MSC 1 1 1. 

Figure |5l shows examples of malformed MSCs: in TO5 exist x,y £ Xm^ \ Eq 
with X ^ y and cr = U-.pa ■ 7p_rb G lin(a;) n \in{y) ^ 0, and hence WFi is violated. 
Indeed there exist x, y with that property such that Vm^ (x) — pass and Vm^ (y) = fail. 
Taking into account the fact that a test system will only observe a it is clear that 7715 
does not describe testable behaviour — which verdict should a test system assign after 
observing cr? The MSC mg is malformed because it violates WF2: e is a choice point 
for the actions lr,pa^, lr,p0'2^ !p,rC. In its initial configuration, the test system can 
either (deterministically) send ai or 02, but will then not be able to test the behaviour 
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Figure 5: Two malformed MSCs 



of the SUT that corresponds to the respective other choice. Also, what happens if 
the test system decides to perform action lr,pa-^ while the SUT, before it has received 
ai, performs !p_rC? This behaviour is not defined by the MSC. For an example of a 
well-formed MSC, the reader may verify that ms from Fig.|2]is indeed well-formed. 



3 Test Case Validity 

We now define the validity of a test case w. r. t. a well-formed test purpose. Our defi- 
nition is different from the available conformance relations for labeled transition sys- 
tems because it assigns different roles to test case and SUT. We show that the well- 
formedness conditions on MSCs from the previous section suffice to guarantee the ex- 
istence of a valid test case. Moreover, we give a simple decision procedure that decides 
validity of a test case and prove its correctness. 

First, we need to formalize the notion of a test case. Intuitively, a test case inter- 
acts with the SUT by means of exchanging messages and finally assigning a verdict. 
Formally, we model a test case as follows: 

Test Cases. A test case is a partial function T : E* ^ Sj, U {6} U V{, where (5 is a 
symbol that denotes quiescence of the test case. 

A run of a test case T is a sequence (Tq, cti, . . . , cr„, of words from E* such that 
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an = £, and Ci — > <Ji+i for ^ i < n, where the relation — > is defined by 

a —.aa <^df ^(o-) defined & (T(ct) = a G EJ, V T(cr) = 5 & a £ ) 

Arun (Toi ci, ■ ■ ■ , cr„ is called com;?Zefe if T((J„) G Note that it is indeed impossible 
to extend a complete run due to the definition of — >. 



In the following we will show how to model test case validity as a certain language 
inclusion problem. 

Test Languages. Both the runs of a test case and the tester observable traces of a 
well-formed test purpose naturally induce test languages, i. e., languages L C E* 
together with a verdict assignments vi^: 

For a test case T, the test language {L-j, vy) is defined by L-j =(if {cr G E* : e — >* 

u} with verdict assignment V'j defined by 



For a well-formed test purpose M, the induced test language (Lm, vm) is defined 
by setting Lm. —df obs(XM:) and, for a G -Lm, vm{<^) =df um{x) for the (due to 
|WFi| uniquely defined) x G X-j^i with a G obs(a;). 

It can easily be shown that vj and vm. are well-defined and satisfy the requirements 
imposed on verdict assignments. 

What is the correct relation between (£^5 ^m) and (L^, vj) to define validity of 
T w. r t. M? Clearly, vj^ and V'j should agree on Ljvt ^ ^t- But what is the right 
relations between Ljvt and Ljl None of the "obvious" choices leads to a satisfactory 
notion of validity: 

• if we would require Ly[ C L-j then there would be no valid test cases for any test 
purpose that allows (inessential) choice between two actions a, 6 G EJ, because 
Ljvt contains traces for both choices while a deterministic test case would be 
limited to only a single choice. 

• requiring Lj C L-]^ would allow the test case to send arbitrary messages to the 
SUT even though these would not be specified in the test purpose 

• if we require Lj n Ljvt ^ then the test case would only be required to react to 
one of the possible many (essential) choices that the SUT might have. 

While the first option matches the intuitive meaning of test case validity best, it 
needs to be modified to eliminate the influence of inessential choice. This is done by 
means of the following equivalence relation on strings: 

Let L C E* be a language. We define an equivalence relation C L x L by set- 
ting a ~L p <^df /O is a permutation of a such that a \T/^ — p \ E^. The equivalence 
class w.r. t. ~i of CT G i is denoted by [aji =df {/? G i : ct ~l p}. 
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\alid{tesi_purpose M; testjcase T; string p) { 
if T(p) ;.v undefined then fail; 
if T(p) G Vf & T(p) =^ vm(p) tlienfail; 
else if T(p) = 6 then 

if eM(M, p) n E- = then fail; 

else foreach a e en(M, p) fl do valid{M, T, p • a); 
else if p • T(p) ^ Lm then fail; 
else valid (M, T, p • T(p)); 
success; 

} 

where en(M, p) =df {a € So : p • a € obs(X3vi;)} 

Algorithm 1 : Validation algorithm. 

Test Case Validity. Let M be a well-formed test purpose and T be a test case for M. 
Then T is called a valid test case w. r. t. M if 

• for every a S Lq- n Lj^, V'j{v) = vj^{v), and 

• for every a e Ljvt with wm(o') G Vf, [(t]lm n Lg- =j= 0- 

Since we have given the definition both for well-formed test purposes and test case 
vaUdity, it would be futile to use one to justify the other. What can be shown formally 
though, is that these notions are compatible in the following sense: 

Theorem 1 Let Mbe a well-formed test purpose. Then there exists a test case T that 
is valid w. r. t. M. T can he computed effectively from M. 

Also, it is easy to see that there are MSCs that violate one of the well-formedness 
conditions, for which no vahd test case exists. 

Deciding Validity. In the following we present an algorithm that decides vaUdity of 
a test case w. r. t. a well-formed test purpose and establish the algorithm's correctness. 
Interestingly, the algorithm does not require the calculation of the -classes but 
only refers to obs(X3v[), ujvt. and Lj^, which can easily be derived from M. 

Theorem 2 Let M a well-formed test purpose and 7 a test case for M. Then T is valid 
w-KtM iffvalidi^M, 7, s) does not fail. 

4 Practical Considerations 

The previous sections have discussed formally the relationship between a test purpose 
defined using MSC and a test system that implements the test purpose. No assump- 
tions have been made on the test system besides that it is deterministic and that it has 
observable test events and a final verdict status. An MSC based validator tool has 
been designed and developed within a joint project between Nokia Research Center 
and Fraunhofer FOKUS. 
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The validator is designed to run against any test system that provides some basic 
functionality, like starting of a test case, retrieving the status of the final verdict, send- 
ing and receiving messages, etc. The basic idea was to create a validator that is not 
only able to validate the abstract test suite but also a real test system (tester), i. e., an 
abstract test suite plus its execution environment plus the glue that is necessary to tie 
the test suite to the actual system under test. Since this glue can be of considerable 
complexity, e.g., consisting of implementations of various protocol stacks, message 
en- and decoders, possibly tailored hardware, etc., testing of the whole test system is 
indeed an important aspect. 

This is also one of the advantages of our approach as compared to other approaches 
like an isolated verification of the abstract test suite or an automatic generation of test 
cases from test purposes. 

Given a sufficiently detailed specification of the test purpose, a combination of 
automatic generation of test cases 1 14 1 from the test purpose together with a validation 
following our approach seems optimal. The validation guarantees both the correctness 
of the implementation of the generation algorithm and of the additional components 
that make up the test system. 

The design of the validator aims to make it as independent of the test system as 
possible by defining a small, well-defined interface to connect the validator to the test 
system. In our case study we have used a TTCN-3 test system with the MSG validator. 
The validator accesses the test system at its (proprietary) control interface to trigger 
the execution of testcases and retrieve the final verdict. It uses TTGN-3's standardized 
communication interface toward the SUT 1 13 1 to exchange messages with the test sys- 
tem. The MSG validator has been implemented using JAVA and the test system runs 
independently of the validator. Although not all work within this project has been com- 
pletely finished, results so far show that using MSGs as test purpose definition language 
and as basis for the test case validation can improve the quality of test cases and thus 
the quality of system implementations. 

5 Future Work 

This paper defines a novel approach to test case validation and provides the necessary 
theoretical background. Yet, it is only a first step toward a working test case validation 
system. In particular, the following issues need to be addressed in the future: 

Algorithms and Complexity. Deciding well-formedness of an MSG M so far re- 
quires the calculation of the semantics X-j^, which is a costly operation. A syntactic 
characterization of well-formedness would be desirable because it would probably al- 
low for faster tests for well-formedness that could, e.g., also be built into an MSG editor 
to support test purpose development by pointing out problematic constructs. Addition- 
ally, a detailed analysis of the complexity of well-formedness and test case validity 
would be desirable. 

Data. Since its last revision, data is an integral part of the MSG language. An exten- 
sion of our approach that also takes into account data passed in messages is essential 
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for the practical applicability of our approach to a wider class of test cases. While this 
should not impose any theoretical problems, it will be a challenge to integrate data into 
the implementation in a user-friendly manner 

Time. MSCs allow to express various timing constraints and timing aspects are im- 
portant in many testing efforts. Therefore, we plan to extend our approach to MSCs 
with timing constraints. From a theoretical point of view, this is probably the most 
interesting way to continue the work presented in this paper. 

6 Appendix 

This appendix contains the proofs of Theorems[nand|2l(for technical reason in reverse 
order). In the following, let M denote a well-formed test purpose. 

From property | WF 1 1 we get that the function : obs(X3vt) Xjyi that maps 
every a € obs(X3v[) to a a;^ G such that a E lin(a;cr) is in fact a well-defined and 
total. It is easy to show the following property: 

Lemma 3 Let Jd be a well-formed test purpose and p, cr G obs(X3vt) with p ^ cr. 
Then (p}^f ^ {a)j^. 

Let M be a well-formed test purpose, T a test case for M and a £ with 
vm(<^) =1= none. A validation for is a complete run pn — > Pi — > ■ ■ ■ — > Pn 

T T T 

such that Pn ~Lm a- and vjipn) = wm(ct). 

It is easy to see that validity of a test case w. r t. a test purpose can equivalently be 
formulated as follows. 

Lemma 4 Let JA be a well-formed test purpose and T a test case for M. Then T is 
valid w. K t. M iff every a g Lj^ has a validation. 

We will need the following technical lemma: 

Lemma 5 Let M. be a well-formed test purpose, a,b £ Yiq be actions, and p,(J £ S*. 
Moreover, assume pa, ph £ Lj^, vj^{a) ^ none, and pb ^ a. If a ^ or & ^ Yi'^(or 
both), then there exists a a' £ Lj^ with a —Lm — VMi^^'), po- ^ cr', and 

Proof 1 Let a, 6, p, a as required by the lemma and let x = {p)j^, and z = {o')j^. 
Let VaTUb G Xjyi \ So such that — x a and i/f, — x o^. b. From [WF/I we get 
X ^ Hi, ^ z. From IWF2I olso x ^ ^ z holds. Hence, there exists u G Xj^ \ So 
with u = xOcaOcband u ^ z and we obtain a' setting a' = pabrj, where rj is the string 
that can be appended to pb to obtain a with the first occurrence of a deleted. From what 
have said before, a' is a linearization of z, hence a —Lm and VM.{a) — vj^{a'). 

Proof 2 (Proof of Theorem|3 As sume that valid(M, T, e) does not fail and let a G 
Ljrt with vm:{o') =^ none and \a\ = n. By Lemma^ it suffices to show that there 
exists a validation of a. To this purpose, we will construct sequences po, . . . , pn and 
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ao, . . . , cr„ such that \pi\ = i, valid(M..'T, pi) is called during the execution of the 
algorithm, pi ^ Ui, Ui —Lm f, and vj^{a) — vj^{ai), for each ^ i ^ n. 

We start with po = e and Gi = a, which satisfies all the required properties. 
Assume that the sequences have been constructed up to i. Since valid{JA, T, pi) does 
not fail, Pi G Lj^ holds and there are the following possibilities: 

• T{p) G Vf & T{p) — vm{p)- In this case, \p\ = n must hold because otherwise 
VMi^'i) =1= none and vj^{pi) =\= none, together with pi < ai, would be (by 
Lemma]3\ a contradiction to the fact that vjyi is a verdict assignment on ijvt- 

• T(p) — S en(M,p) n 4= 0. Then i < n must hold and since pi < ai, 
there exists b G Ec{T) such that pib ^ Ui. If b ^ "E^ then there will be a 
call valid(M, 7, pib) and we set pi+i —dt Pib and (Ti+i =df to continue 
the sequences. Clearly, this satisfies all necessary properties. If b E T,'^ then 
let a G en(M, pi) H S^. Then a, 6, pi, ai satisfy the prerequisites of Lemma]5\ 
which yields the existence of a'l G Lj^ with ai '^'i' ^M(o'i) — W]v[(o'^), and 
Pia ^ a'i. If we set pi+i —df then we have extended the sequence as required. 

• 7{p) — a (z Ti^'^ &L pa (z Lm- ^« this case we necessarily have to set Pi+i —di 
Pia and we need to show the existence of a suitable CTi+i. This can be done 
similar to the previous case using Lemma^ 

It is easy to see that 7{pn) G Vi and by construction it holds that pn —Lm '^^ 
well as V'j{pn) = vjaW)- Moreover, obviously Po ^ ■ ■ ■ ^ Pn is complete run and 

thus we have found the desired validation for a. 

For the converse direction, let 7 be a test case that is valid w. r. t. M. We need to 
show that the call valid{JA, T, e) does not fail. Hence assume that is does fail and let 
p G Lj^i a prefix-maximal word such that valid{JA, 7, p) is evaluated. By definition of 
valid, pinLj^ must holds. One of the following choices for p is the one that leads to 
failure. 

• 7{p) is undefined, then obviously, for every a G Lj^ with p ^ a, [a]^^ fl L-j = 

0. 

• 7{p) G Vf and 7{p) ^ vm which violates the first condition in the definition 
of test case validity. 

• 7{p) = 6 and en{Jvl, p)riY.li — 0. Ifvj^{p) none then L-j and Ljy[, then again 
the first condition of the definition of test case validity is violated. If vm{p) = 
none then there exists a G Ljvt with vm (c) =1= none and p < a. It is easy to 
see that, if there exists 77 G [cJlj^j H Lj, then p < i], but since 7{p) = 6 and 
en (M, p) n = 0, for every x G with p < X, X $ holds and hence the 
does not exists such a rj. 

• The case that 7{p) = a G Sj, but p ■ a ^ Lj^ is analog to the previous case. 
Hence, valid{Jd, 7, e) cannot fail. 
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Proofs (Proof of Theorem^) We define a test case 7 as follows: for a G obs(Xjvi;); 

r WM((a;)3vt),i/en(M,a) = 0; 

y a, for an arbitrary a G e«(M, cr) Pi otherwise 

It is easy to see that for this test case 7, valid{JA, T, e) does indeed not fail and 
hence, by Theorem^ 7 is a valid test case w. r. t. M. 
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